High-speed Cluster Interconnects Myrinet, Quadrics, and Infiniband Achieve Low Latency and High Bandwidth with Low Host Overhead. However, They Show Quite Different Performance Behaviors When Handling Communication Buffer Reuse

Today’s distributed and high-performance applications require high computational power and high communication performance. Recently, the computational power of commodity PCs has doubled about every 18 months. At the same time, network interconnects that provide very low latency and very high bandwidth are also emerging. This is a promising trend in building highperformance computing environments by clustering—combining the computational power of commodity PCs with the communication performance of high-speed network interconnects. There are several network interconnects that provide low latency (less than 10 μs) and high bandwidth (several gigabytes per second). Two of the leading products are Myrinet and Quadrics. Recently, InfiniBand has entered the high-performance computing market. All three interconnects share similarities. For one, they provide user-level access to network interface cards for performing communication; they also support access to remote processes’ memory address spaces. However, they also differ in many ways. So the question arises: How can we conduct a meaningful performance comparison among all three interconnects? Traditionally, researchers have used simple microbenchmarks, such as latency and bandwidth tests, to characterize a network interconnect’s communication performance. Later, they proposed more sophisticated models such as LogP. However, these tests and models focus on general parallel computing systems and do not address many features present in these emerging commercial interconnects. Another way to evaluate different network interconnects is to use real-world applications. However, real applications usually run on top of a middleware layer such as the Message Passing Interface (MPI). Therefore, the application-level performance reflects not only the capability of the network interconnects, but also the quality of the MPI implementations and the design choices of the MPI implementers. Thus, to provide more insight into Jiuxing Liu Balasubramanian Chandrasekaran